1. Vehicle Exterior Monitoring
1.1 General
During the process of operating a motor vehicle, it is necessary for the operator to obtain information concerning the proximity of various dangerous objects and their relative velocities for the operator to make sound driving decisions, such as whether or not there is enough time to change lanes. This information should be obtained from the area that completely surrounds the vehicle. In order to gather this information, the operator is frequently required to physically turn his or her head to check for occupancy of a blind spot, for example. In taking such an action, the attention of the driver is invariably momentarily diverted from control of the vehicle.
For an automobile, the blind spots typically occur on either side of the vehicle starting approximately at the position of the driver and extending backwards sometimes beyond the rear of the vehicle. The locations of these blind spots depend heavily on the adjustment of the angle of the rear view mirror. Different areas are in the blind spot depending on the mirror angle. Since it is in general not known whether or how the mirror is set for the particular vehicle, a blind spot detector must detect objects anywhere along the sides of the vehicle, and even behind the vehicle, regardless of the mirror setting.
The problem is more complicated for trucks, enclosed farm tractors and construction equipment that not only can have much larger blind spots along the sides of the vehicle but also can have a serious blind spot starting in front of the right front bumper of the vehicle and extending beyond the right door. This blind spot is particularly serious with trucks and even vans, SUVs and cars in urban driving where small vehicles, motorcycles, pedestrians, bicycles etc. in this area can be completely hidden from the view of the driver.
Several systems have been designed which attempt to rotate the mirror to pick up or allow a driver to visually see the object in the blind spot. This is difficult to do without knowledge of the location of the eyes of the driver. For most systems that do not incorporate an occupant sensor capable of determining the location of the driver's eyes, there is a risk that the mirrors will be positioned wrongly thus exacerbating rather than helping the blind spot detection problem. Also, a system that rotates the mirror will make the driver nervous since he or she will not be able to see the scene that he or she is accustomed to seeing in the mirror.
Monitoring systems that are based on radar or ultrasound have been available but not widely adopted for automobile blind spot detection for reasons related to cost, accuracy and false alarms. Both systems use beams of energy that can become several feet in diameter by the time they reach the edges of the blind spot and thus can confuse a large vehicle or a guardrail, sign, parked car etc. two lanes over with a vehicle in the blind spot. Some such systems attempt to filter threatening objects from non-threatening objects based on the relative speed of the object and thus err by eliminating a significant number of such threats. A tradeoff exists in all such systems where, if all threatening objects are made known to the driver, the false alarm rate becomes unacceptable and the driver soon loses confidence in the system and ignores it. If the false alarm rate is kept low, many dangerous situations can be ignored.
These prior art systems thus have serious failure modes. The lesson is that if a vision-based system such as the rear view mirror is going to be replaced with a non-vision system, then the non-vision system must be almost as good as the vision system or it will not be adopted.
Some other problems arise when a vehicle strays into the lane of the host vehicle, i.e., the vehicle with the blind spot detector. Most systems will fail to warn the operator and thus an accident can result. As such, the blind spot problem is really two problems relating to the motion of the potentially striking vehicle and the potentially struck vehicle.
A problem that is addressed herein is to determine what information is needed about the object in the blind spot and then the manner in which this information is presented to the vehicle operator so as to eliminate accidents caused by the failure of the operator to see such an object. This information includes the accurate location of the object relative to the host vehicle, its size, its relative and/or absolute velocity, and the identity or kind of object. This information must be known regardless of the changes in road geometry such as steep hills and sharp curves or changes in environmental conditions. Naturally, the system must be low cost if it is going to be purchased by the public or installed by vehicle manufacturers.
Studies have shown that giving the driver an extra half-second could eliminate as many as 50 percent of the accidents. Thus, the risk of an accident must also be communicated to the operator in a timely fashion to permit the driver to take evasive action or not take a particular action such as a lane change.
What is needed therefore is a system that acts like the eyes of the driver and interprets the situation and only gives a warning when there is a real possibility of an accident. A passive warning can be given in the form of a light on the mirror whenever an object is in the blind spot; however, an active signal such as an audible signal or an intervention in the steering of the automobile should only be provided when it is necessary to prevent an accident. This system must work with very high reliability and accuracy since the consequences of an error can be serious injuries or death.
One approach reported in the paper C. Thorpe et al., “Driving in traffic: Short range sensing for Urban Collision Avoidance”, Carnegie Mellon University (CMU), January 2002, is a relatively superficial discussion based on the use of vision, radar, ladar and other systems for interrogating the environment around the vehicle. No mention is made of how various objects are identified that could pose threats for vehicular occupants or pedestrians or the distinction between objects that may temporarily occupy a space from those that are permanently part of the infrastructure. The technology described below and in other patents assigned to Automotive Technologies International (ATI) and Intelligent Technologies International (ITI) presents in detail how such objects are found and identified and how the location of fixed objects, such as curbs, are known and are part of a vehicle resident accurate map. The CMU solution, based on observations without the aid of location determining technologies such as DGPS and accurate maps, will require reliance on models of vehicles, models of pedestrians and human factors, the analysis of all of which is at best inexact and incapable of solving the problem of collision avoidance. Also, by not having a good identification of such objects, such as provided herein, the CMU solution will not be able to provide the proper response in critical situations. The inventions described below and in the related patents and patent applications of ATI and ITI, on the other hand, solve the total problem of avoiding fatalities on roadways and lead toward Zero Fatalities. The system and method for achieving this objective is referred to with the trademarks ZERO FATALITIES™, ROAD TO ZERO FATALITIES® and RTZF®.
The use of range gating is a significant part of several implementations of inventions disclosed herein. The concept of range gating is currently not new when used to determine to distance to a point on an object that is illuminated by radar, for example (see U.S. Pat. No. 3,735,398 which describes use of range gating with an ultrawide band radar system). The use of range gating in conjunction with acquiring an image of an object and allowing separation or segmentation of the object's image from reflections from other objects that are at different distances from a vehicle is believed to be unique to the inventions disclosed herein.
1.2 Blind Spot Detection Systems
The term “blind spot” as used herein is meant to include more than the common definition of the term. See section 8. Definitions for a more complete definition.
In U. Dravidam and S. Tosunoglu, “A survey on automobile collision avoidance system”, Florida conference on recent advances in robotics 1999, the authors provide a good review of the field of obstacle sensors. What follows is a summary of their analysis. Obstacle sensors such as used for blind spot detection can be divided into three types:
Optical sensors include passive infrared, laser radar and vision. They generally are sensitive to external environmental conditions, which may not be a problem for blind spot detection since the objects to be detected are usually nearby the host vehicle. Passive infrared and vision cannot provide a direct measurement of distance to an object unless part of the field of view is illuminated by a point or structured light. Laser radar does provide the capability of direct distance measurement, as will be described below, and a stereo camera can also provide distance information.
AMCW (amplitude modulated continuous wave), FMCW (frequency modulated continuous wave) and impulse and noise or pseudo-noise (CDMA—code modulated multiple access) radar are not generally affected by adverse environmental conditions. Although relatively expensive, FMCW radar is a good technique for long-range distance measurement provided the object to be measured can be separated from other objects. Radar in general has a high false alarm rate due to the large pixel size at any significant distance from the host vehicle, to multipath effects and reflections from signs, bridges, guardrails etc.
Ultrasonics are good in applications where only short relative distance measurements are required, since they are able to provide high distance to the target resolution for a relatively low cost. However, for imaging applications, the slow speed and relatively large pixel size renders ultrasonics marginal even for close up targets. Also, ultrasonic waves can be significantly distorted by thermal gradients and wind.
Various researchers have attempted combinations of these technologies with the particular combination of laser radar and pulse or FMCW being quite advantageous for long distance collision avoidance applications.
What follows is a brief description of the principles of operation for different types of sensors including their main advantages and disadvantages. For blind spot applications, sensors should be able to accurately determine the location of the object and the speed of the obstacle relative to the host vehicle. How well this is achieved can be measured with the following indicators:
Sensing range: the maximum and minimum range over which the technique can be used.
Range Resolution: the relative change in range that can be measured.
Pixel Resolution: the width of the beam or size of the pixel received and to which the sensor is sensitive.
Response time: how quickly the sensor can respond to a change in the blind spot occupancy.
Ultrasonics: These sensors work by measuring the time-to-flight of a short burst of ultrasound energy typically at a frequency of 30-200 kHz. The time taken for the ultrasonic waves to travel to and return from the obstacle is directly proportional to the distance between the obstacle and the host vehicle. The main advantage is their relative low cost and small size. These sensors are also very sensitive to changes in the density of air that can be caused by, e.g., high wind velocity and temperature gradients. Velocity can be measured by the Doppler frequency
Passive Infrared: These sensors measure the thermal energy emitted by objects. Their main advantage is their low cost and small size, and main disadvantage is their inability to determine the distance to a detected object and slow response time.
Laser Radar: As with regular radar, two techniques exist: (1) a pulsed-beam of infrared light coupled with time-of-flight measurements, and (2) the modulation of a continuous light beam. The pulsed technique offers long range, high directionality, and fast response time. Its limitations are its sensitivity to environmental conditions.
FMCW or AMCW Radar: This type of radar uses modulated microwave or millimeter frequencies, so that the frequency difference between the reflected signal and the transmitted signal is proportional to the relative velocity of the object. When two waves of slightly different frequencies are used, the distance to the object can also be determined by the phase relationship between the two received reflections. Despite its high cost, this technique offers the advantages of being insensitive to environmental conditions, but the disadvantage of having a large pixel size. Velocity can be measured by the Doppler frequency shift.
Impulse Radar: This radar differs from FMCW in that it uses very short pulses instead of a continuous wave. Like FMCW radar, it is insensitive to environmental conditions, and the cost is significantly lower than FMCW. Distance can be determined by time-of-flight measurements and velocity can be determined from successive distance measurements. It also has the disadvantage of having a large pixel size resulting in a high false alarm rate and too little information to permit object identification.
Capacitive and Magnetic: Capacitive and magnetic sensors are able to detect close objects (within about 2 m.), using the capacitance or magnetic field variations between electrodes excited at low frequencies, typically about 5 kHz. Despite their limited range, they are low in cost, and robust to external environmental effects. Poor resolution compared to other techniques makes it unlikely that these devices will be used for blind spot detection since most objects are close to the vehicle.
Vision Systems: These techniques are based on the use of a camera and image-processing software. They are sensitive to external environmental conditions; however, this is not a significant shortcoming for blind spot detection. Active infrared vision systems can have significantly longer range in smoke, fog, snow and rain than human eyesight. This is especially the case if range gating is used (see U.S. patent application Ser. No. 11/034,325 filed Jan. 12, 2005, now U.S. Pat. No. 7,202,776).
Considering now some relevant patent prior art. U.S. Pat. No. 4,766,421; U.S. Pat. No. 4,926,170; U.S. Pat. No. 5,122,796; U.S. Pat. No. 5,311,012; U.S. Pat. No. 5,122,796; U.S. Pat. No. 5,354,983; U.S. Pat. No. 5,418,359; U.S. Pat. No. 5,463,384 and U.S. Pat. No. 5,675,326 and International Publication No. WO 90/13103 are all assigned to Auto-Sense, Ltd., Denver, Colo. and describe modulated optical systems. However, these references do not disclose a camera and in fact, each receiver is a single pixel device. The sensor is not mounted on the side rear view mirror but instead is mounted on the rear of the vehicle. These references disclose the use of multiple detectors and thereby achieving a sort of mapping of the detected object into one of several zones. The references also provide a crude velocity measurement of the object moving from one zone to another. Otherwise, they do not provide accurate ranging.
These references describe a blind spot detection system wherein beams of infrared radiation are sent from the interrogating or host vehicle at a significant angle in order to illuminate possible objects in an adjacent lane. No direct measurement of the distance is achieved, however, in some cases multiple detectors are used in such a way that when the adjacent detected vehicle is very close to the detector, that is, below the threshold distance, the sensing of the adjacent vehicle is suppressed. In other cases, multiple beams of infrared are used and distance is inferred by the reception of reflected radiation. The detectors are single pixel devices. No attempt is made to image the detected object. Also, no attempt is made to directly measure the location of the detected object.
U.S. Pat. No. 5,008,678 describes a phased array radar system wherein the antenna can be made to conform to the geometry of an edge of the automobile. The locations of the antenna, however, make it difficult to detect many objects in the side blind spots. The particular location and velocity of such objects are also not accurately determined. No image of the device is formed. The device is based on a single pixel having a relatively large size making recognition and identification of the object impossible.
U.S. Pat. No. 5,087,918 describes the use of a combination of two types of radar: dual frequency Doppler radars and frequency modulated continuous wave radar (FMCW). The system provides an indication of the range of the object from the vehicle but does not indicate where in a plane perpendicular to the vehicle the object is located and therefore whether it is a threat or not. Also, the system does not apply pattern recognition so that different types of objects in the blind spot can be identified. This patent gives a good description of the limitations of radar systems.
U.S. Pat. No. 5,229,975 describes a method for diagnosing when the system is not operating properly by placing an LED outside the vehicle next to the sensor. This is a single pixel device and thus no imaging or object recognition is possible. Range is not measured directly but through a series of sensors whereby each sensor covers a particular zone. Thus, no accurate range measurement is provided. As the object moves in the blind spot area, it is sensed by a variety of the sensors and the last one to sense it gives a crude indication of the distance.
U.S. Pat. No. 5,235,316 describes an ultrasonic blind spot detecting system that in fact interrogates as much as 200 degrees around the vehicle. It is mounted in place of the conventional mirror and a new side mirror is provided. The ultrasonic sensor rotates until it locates an object and then it causes the mirror to rotate so that the driver can see the object. The patent does not take an image of the threatening object or the object in blind spot. It is a one-pixel device and it does not employ pattern recognition. Additionally, it provides too much information for the driver thus creating the possibility of driver information overload.
U.S. Pat. No. 5,289,321 describes a camera and an LCD display on the instrument panel. The camera views rearward and the driver sees the image captured on an LCD. It does not disclose a camera mounted on the rear view mirror. The main problem is that the LCD driver-viewing screen is more likely to confuse than to aid the driver due to its poor dynamic light intensity range and the ability to relate the image to the location and velocity of the object in the blind spot.
U.S. Pat. No. 5,291,261 describes illumination ports at an angle with respect to single pixel receiver ports. Fiber optics are used to transmit the few pixels to a central processing station. There is no direct ranging. Some crude ranging is accomplished since when the object is in certain zones where the projected light overlays the receiving fields, the reflected light can be sensed. It requires multiple locations and cannot be mounted, for example, on the side rearview mirror.
U.S. Pat. No. 5,325,096 uses Doppler radar to determine the presence and relative velocity of an object blind spot. It filters out stationary objects and concentrates only on those objects that have approximately the same velocity as the vehicle. As a result, many objects, such as a high speed passing vehicle, are missed. A light is used to indicate the presence of an occupying item in the blind spot area and an audible alarm is sounded when the turn signal is activated. There is some crude range measurement possible. It is also a single pixel device and thus, no image of the object can be formed. It invariably will miss objects that move rapidly into blind spot. There is no precise ranging. It does not appear that the system can be easily adjusted for vehicles of different length.
U.S. Pat. No. 5,424,952 describes an optical system using cameras wherein distance is measured stereoscopically. Objects that are not in the adjacent lane are ignored. The problems are that no attempt is made to analyze the image or to determine its velocity and therefore, a high false alarm rate can be expected. Although the image is captured, the information is ignored except for its use to determine a stereo distance.
U.S. Pat. No. 5,467,072 describes a phased array radar system that can scan the blind spot as well as all other areas around vehicle. However, the system does not provide an image and therefore no optical pattern recognition is possible. The 10-degree divergence angle of radar indicates that a single pixel has a diameter of over 3 feet at 20 feet from the radar transmitter, which is insufficient resolution to determine the lane that the threatening vehicle is occupying, especially if there is a slight curvature in the road. Such a system is not sufficiently accurate to provide drivers who are attempting to merge into adjacent lanes with sufficiently accurate position information to permit a safe merge under heavy traffic without visual contact. Additionally, there is no pattern recognition claimed or even possible with this low resolution device.
U.S. Pat. No. 5,517,196 describes a multi-frequency radar system using Doppler techniques. Stationary objects are filtered out. In fact, the system also only looks at objects that are traveling at approximately the same speed as the host vehicle. It has a good range of 0.25 to 100 feet. Some problems are that this system will interfere with other vehicles having the same system. There appears to be no direct measurement of the object's position, but it does give a good distance resolution of 0.55 feet. This patent also contemplates the use of steering wheel angle and vehicle speed inputs to the system. Even though ultrasonic, infrared and radar are disclosed, it is still a single pixel system. Once again, the system will invariably miss a high-speed vehicle passing on either the right or the left since it is limited to a two mile per hour velocity difference between the blind spot object and the host vehicle. It also appears to be a very expensive system. Another potential problem is that when an especially long truck having the system of this patent is turning, the system would pick up the end of truck and treat it as an object in the blind spot.
U.S. Pat. No. 5,668,539 uses thermal imaging to recognize a car or truck in the blind spot. It uses a vibrating element between the field of view containing the blind spot using three lenses thus giving three different locations and a reference field of view that is the road behind the vehicle. One problem with this device is that this system does not know where the infrared rays are coming from. It could be from the sun or from reflections from the wrong lane. The slow cycle time prevents averaging to eliminate errors. At a 60 km per hour passing rate, the vehicle will travel 1.7 m each cycle based on a 10 hertz cycle rate. The patent also mentions that the form of the signal that comes from a vehicle and the blind spot has high frequency associated with it whereas the form of the signal from the road does not. This is an alternate method of discriminating between a vehicle and the road but one that still lacks resolution.
U.S. Pat. No. 5,670,935 describes a camera and a display where the actual images of the vehicle in the blind spot and behind the subject vehicle are displayed on the visual display. Unfortunately, the various figures in the patent that illustrate this phenomenon are not accurate and appear to show that the positions of the vehicles relative to the subject vehicle can be visually seen which is not the case. Thus, the invention described in this patent cannot be used for blind spot detection in the manner described since the relative locations of vehicles cannot be determined. Also, no attempt has been made to identify and analyze objects in the blind spot and warn the driver of a pending accident. U.S. Pat. No. 5,765,116 describes a system wherein a torque is artificially applied to the steering wheel to keep a driver in the center of his lane. This is not a blind spot related patent but this same technique can be used to prevent a driver from attempting to change lanes when there is an object in the blind spot.
U.S. Pat. No. 6,038,496 describes a lane boundary finder. It uses a linear array of LEDs plus a linear CCD with a total of 64 pixels in the CCD array. It can be used for blind spot monitoring, although this is not the main purpose of this invention. The CCD array suffers from the problem that, due to its limited dynamic range, it can be overwhelmed by light from the sun, for example, reflected off a vehicle or other surface. Since there is only a linear array of only 64 pixels, no information as to what is in the blind spot can be obtained. In other words, the system knows that something is in the blind spot but does not know what it is or even accurately where it is. Nevertheless, the use of the scanning system disclosed wherein the particular pixel or the beam that is being activated to create a light on a leading or reflecting surface is an important addition to the technology and may also be used with this invention.
U.S. Pat. No. 6,501,371 describes a method for locating eyes of a vehicle driver and locating an object external to the vehicle and adjusting a rear view mirror so that the driver sees the external object. All of these ideas are believed to have previously been disclosed in patents assigned to ATI and ITI.
International Publication No. WO 95/25322 describes a passive infrared blind spot detector that processes infrared waves based on a crude form of pattern recognition. There is no accurate ranging and there will likely be a high false alarm rate with this system. There is also sometimes a period when the system is unavailable due to changes in ambient conditions such as the start of a rain shower or when the temperature of the road changes due to shading. It is a one-pixel device and therefore does not permit the location of the object in the blind spot to be determined. This device and other similar passive infrared devices will have trouble distinguishing between a small objects such as a motorcycle which is relatively close to the sensor and larger objects such as a truck which are relatively far away, for example two lanes over. As a result, it will likely falsely indicate that a relatively large object is within a danger zone when in reality the object is at a distance and does not pose a threat.
International Publication No. WO 99/42856 describes a rear of vehicle mounted blind spot detector based on various radar systems. It has the capability of tracking multiple targets and of accurately determining the ranges to the various targets using range-gating techniques. It does not attempt to capture an image of an object in the blind spot or determine the identity of such an object and thus many non-threatening objects will appear to be threatening. Accordingly, the system can be expected to have a high false alarm rate.
In general, the poor resolution of radar systems requires that they use relative velocity as a filter in order to reduce the false alarm rate. As a result, such systems miss a high-speed vehicle that is in the blind spot and was not observed approaching the blind spot by the driver. This is a very common occurrence on European superhighways and in the United States on two lane roads.
Thus, none of the related art described above discloses a method or apparatus of monitoring the area surrounding a vehicle that analyzes an image of one or more objects that occupy the blind spot, identifying them and determining the location and relative velocity of the objects relative to the host vehicle in a manner that permits an accurate warning to be issued to the driver of a potentially dangerous situation.
1.3 Optical methods
Optics can be used in several configurations for monitoring the exterior of a vehicle. The receiver can be a CCD or CMOS imager, to receive the emitted or reflected light. A laser can either be used in a scanning mode, or, through the use of a lens, a cone or beam of light can be created which covers a large portion of the object in the blind spot. Alternately, a combination of these techniques can be used such as a scanning beam or an adjustable lens system that converts a laser beam to a converging, constant diameter or expanding illuminator. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest on the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in U.S. Pat. No. 5,653,462.
In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to illuminate a desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. The cost of CCD and CMOS arrays has dropped substantially with the result that this configuration is now a cost-effective system for monitoring the blind spot as long as the distance from the transmitter to the objects is not needed. If greater distance is required, then a laser system using modulation and phase detection or time-of-flight techniques, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein, or other equivalent systems, can be used. In a particular implementation, the illuminating light is in the form of a modulated infrared laser light that is scanned in a line that illuminates an object in the blind spot. The reflected light is received by a pin or avalanche diode, or equivalent, after passing through a narrow frequency band notch or other appropriate filter. The diode is a single pixel device, although several or an array of diodes can be used, but since the direction of the transmitted light is known, the direction of the reflected light is also known. The phase of received light is then compared with the transmitted light. The modulating frequency can be selected so that no more than one wavelength of light exists within the blind spot area. The location of the reflecting object can then be determined by the phase difference between the transmitted and reflected light. Although the described system uses a line scan, it is also possible to use a two-dimensional scan and thereby obtain a three-dimensional map of the area of interest. This can be done using a pin or avalanche diode or equivalent as described or the light can be received by a CMOS array and can be monitored on a pixel by pixel basis in a manner similar to the PMD system described in Schwarte, et. al. “New Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technology” 4th International Conference—Advanced Microsystems for Automotive Applications, Apr. 6th/7th, 2000, Berlin (Germany). In this latter case, the entire blind spot area may be flooded with modulated infrared light as described in the paper. On the other hand, it is difficult to overcome the light from natural sources such as the sun by a single floodlight source and therefore a line or even a scanning point source permits better distance measurement using a light source of reasonable intensity. An alternative is to increase the power of the transmitted illumination, for example by using a high power diode laser, and to increase the beam diameter to remain below eye safety limits.
This technique can also be used for vehicle velocity determination and at the same time the topology of the ground covered by the scanning laser can be determined and reflections from rocks and other debris can be eliminated from the velocity calculation for applications where the prime goal is to determine the vehicle velocity relative to the ground.
A mechanical focusing system, such as used on some camera systems can determine the initial position of an object in the blind spot. A distance measuring system based of focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras. Although the Subbarao patent provides a good discussion of the camera focusing art, it can be more complicated than is needed for the practicing the instant invention. A neural network or optical correlation system, as described below, can also be used to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD's and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of some of the inventions disclosed herein. Distance can also be determined by the system described in U.S. Pat. No. 5,003,166 by the spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can also be measured by using time-of-flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching herein.
There will be conditions when the optical system from the CMOS camera has deteriorated due to contaminants obscuring the lens. Similarly, the light emitting laser diodes will emit less light if the lenses are soiled. The system of this invention contemplates a continuous diagnostic feature that will permit sensing of either of these conditions. This can be accomplished in a variety of ways such as a laser diode aimed at the road surface close to the vehicle but within view of the CMOS camera. If the reflection over a period of time is not sufficient, then a warning light will appear on the instrument panel informing the driver that maintenance is required. Naturally, there are many other methods by which a similar diagnostic can be accomplished.
Except as noted, there appears to be no significant prior art for the optically based apparatus and methods of the inventions disclosed herein. In particular for optical systems that obtain sufficient information about objects in the area surrounding the vehicle to permit a pattern recognition system.
In U.S. patent application Pub. Nos. 20020191388, 20030193980, 20030193981 and 20030198271, a method for illuminating a highway is disclosed which permits multiple vehicles approaching each other to pulse illuminate the roadway in a manner timed by a GPS timing signal such that they do not blind each other's imaging system. The imaging system is turned off except during an interval necessary for the illumination to travel to the roadway and return. If groups of vehicles traveling toward each other transmit at different times, when one group is transmitting the other opposing group has its receiver turned off and thus is not blinded. No mention is made of pattern recognition, positioning the display in the field of view of the driver, or measuring the distance to the object of interest and thus the limited use of the inventions disclosed in these patent publications is not believed to anticipate inventions disclosed herein.
1.4 Combined Optical and Acoustic Methods
Both laser and non-laser optical systems in general are good at determining the location of objects within the two-dimensional plane of the image and a pulsed or modulated laser or continuous modulated radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time-of-flight, correlation or by phase measurement. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases, the mere location in the field of view can be used to estimate the position of the object in the blind spot, for example.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z-axis is from the transducer to the object, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in U.S. Pat. No. 5,835,613 and U.S. Pat. No. 5,845,000.
One example of such a system is an optical system that uses natural light coupled with a lens and CCD or CMOS array which receives and displays the image and an analog to digital converter (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to an artificial neural network (ANN), correlation system or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the area or volume of interest. The receiving transducer feeds its data into an ADC and from there, the converted data is directed to the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the object and the phased array can direct a narrow beam to the location and determine the distance to the object through time-of-flight, for example. This technique is especially applicable for objects near the host vehicle. The combination of radar and optics can also be used in a similar manner at a significant cost penalty.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the object can be updated. Second, ultrasound waves are diffracted by changes in air density that can occur when thermal gradients are present or when there is a high-speed flow of air past the transducer, compensation techniques exist as reported in the current assignee's patents and applications such as U.S. Pat. No. 6,279,946, U.S. Pat. No. 6,517,107 and U.S. Pat. No. 6,856,876. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data. In spite of these drawbacks, ultrasound is a fine solution in some applications such as for velocity and displacement determination for automobiles in rear end impacts and farm tractors and construction machines where the operating speeds are low compared with automobiles.
1.5 Discussion of the External Monitoring Problem and Solutions
The above review of related art blind spot detecting systems illustrates that no existing system is believed to be sufficiently adequate. A fundamental problem is that vehicle operators are familiar with visual systems and inherently distrust all other technology. As soon as the non-visual system gives a false alarm or fails to detect an object in the blind spot, the operator will cease to depend on the system. Theoretically, the best systems would be based on cameras that allow the operator to view all of the blind spots. However, there are no adequate display systems that will appear to the operator to be equivalent to an actual view of the scene. CRTs and LCDs require driver concentration and do not have the dynamic range of lighting that is comparable to the real world. Either the display will be too bright at night or too dim during daylight or the wrong object will be bright compared with the object of interest. Although radar systems can accurately measure distance to an object, they are poor at placing the object in the lateral and vertical coordinates relative to the vehicle and thus create many false alarms.
The simplest system must be able to accurately position the object in the blind spot, or other area of interest, relative to the host vehicle and inform the driver that a collision potential exists if the driver decides to change lanes, for example. This warning must be given to the driver either at a place where he can almost subconsciously observe the warning when he is contemplating a lane change maneuver, or it must provide an audible warning if he attempts to make such a lane change maneuver. Finally, such a system might even prevent a driver from executing such a maneuver. A more sophisticated system involving icons will be discussed below.
To accomplish these goals, it is desirable to positively locate an object in the area of interest such as one or more blind spots and provide some identification as to what that object is. A driver will respond quite differently if the object is a guardrail or a line of parked cars then he will if it is a Porsche overtaking him at 150 kph.
Thus, the requirements of the system are to identify the object and to locate the object relative to the host vehicle. To identify the object preferably requires a pattern recognition system such as neural networks or optical correlation systems discussed below. To locate the object preferably requires some means for measuring the distance from the camera or other sensor to the object. A CMOS camera is quite capable of capturing an image of an object in the blind spot, for example, and if the camera is an HDRC camera, then it can perform well under all normal lighting conditions from midnight to bright sunshine especially if minimal illumination is provided on dark nights.
However, even the HDRC camera can be blinded by the sun and thus an alternate solution is to use a scanning laser radar where the point of IR can overpower the emissions of the sun at that wavelength. A scanning laser radar can scan in either one or two dimensions depending on the design. The scanning mechanism can be a rotating polygon mirror, vibrating galvanometer-type mirror, a vibrating MEMS mirror or a solid-state acoustical-optical crystal. In the case of the solid-state device, one or more special lenses or reflectors can be used to increase the effective scan angle. The IR wavelength can be in the far, mid or near IR bands. If the wavelength is in the mid-IR band, it can be selected so as to provide the greatest range in rain, snow or fog. Also, its amplitude at the selected wavelength should be sufficient to be detected in bright sunlight. Range gating can also be used to partially overcome the effects of rain, snow, fog and/or smoke.
An alternate to the HDRC camera is to use an electronic shutter and/or variable iris. In this case, the camera can be operated in the range that best images objects of interest or a series of images can be taken at different settings and portions of each image combined to create a sharp image of the area of interest as reported in S. K. Nayar and T. Mitsunaga, “High Dynamic Range Imaging: Spatially Varying Pixel Exposures”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head Island, S.C., June 2000, for example.
The measurement of the distance to the object can be accomplished in many different ways including ultrasonically, using laser radar (lidar), FMCW or AMCW radar, micropower impulse radar, any of which can be combined with range gating. All of these distance measuring techniques as well as stereographic, focusing, structured light, triangulation, correlation using random or pseudorandom code modulation and other similar techniques are envisioned for use in some of the inventions described herein and in general there is little prior art describing any of these methods or systems for monitoring the area exterior to a vehicle.
A low-cost preferred approach of solving the distance-measuring problem that is consistent with an HDRC camera system is to project onto the volume of the area of interest a series of infrared light pulses. These pulses are created by an array of laser diodes that are displaced from the camera in such a manner that a pulse of light reflected off of an object in the blind spot will appear on a certain pixel area in the camera field of view and since the location of the transmission of the pulse is known and the location of the camera is known, the distance to the reflecting surface is also known by triangulation. By a judicial choice of transmission angles from the laser diode array, the entire volume of the blind spot can be covered with sufficient accuracy so that no significant object can penetrate the blind spot without creating a reflection and thereby permitting the distance to the object to be determined. No prior art has been uncovered describing this or a similar principle.
In one implementation, a series of pulses from a laser diode array are contemplated. Other techniques will also accomplish the same goal, however, at a generally higher cost. For example, a continuous laser beam can be used that would scan the blind spot area, for example, in either one or two dimensions. Since the direction of the laser will be known that all times its reflection and excitation of pixels on the CMOS array would permit, once again, an accurate, mapping of the distance to various points on the object in the blind spot to be accomplished. This technique however requires a scanning laser system that in general, although more accurate, would be more expensive than a simple array of LEDs. Once again, the photonic mixing device described above would also provide a three-dimensional image of the contents of the blind spot as would a similar and preferred system described below. Another technique is to superimpose on the blind spot area, for example, a pattern of light commonly referred to as structured light. The source of the structured light must be displaced from the imaging array. By observing characteristics of the reflected pattern, such as the distances between portions of the pattern, the distance to the object can be determined. This system, although common in machine vision applications, requires greater computational resources than the simple LED array described above. Nevertheless, it is a viable approach and envisioned for use in the invention and again there appears to be little, if any prior art for the use on structured light in monitoring the area surrounding a vehicle.
Various forms of structured light coupled with other patterns which are either inherent in the lens of the camera or are superimposed mathematically on the image can create what is commonly known as Moiré patterns that also permit the determination of the distance from the camera to the object. In some sophisticated examples, this technique can actually provide the equivalent of topographical maps of the object in the area of interest that would be of value in interpreting or identifying an object. However, these techniques require more computational power and may are not be as cost-effective as the simple LED array described above or a linear scanning LED or laser with a pin diode, or equivalent, receiver as disclosed below. Nevertheless, these techniques are viable for many applications and will become more so as component prices decrease.
All of these systems permit differentiation between light that is reflected from the transmitted infrared systems and reflected light from the sunlight, for example. It is quite likely that at certain times, certain pixels in the camera will receive infrared radiation that overwhelms the reflection of the infrared sent by the host vehicle system. If this radiation comes from pixels other than those that are expected, then the system will know that the results are erroneous. Thus, the systems described above have the capability of permitting the diagnosis of the data and thereby achieving a high accuracy of the results. If the results do not agree with what is expected, then they can be ignored. If that happens over a significant period of time, then the operator of the vehicle is warned that the area monitoring system is non-operational.
Using sophisticated image processing and mathematical techniques, however, it is expected that those periods of non-functionality will be minimal. The vehicle operator however will not be subjected to a false alarm but instead will be told that the system is temporarily non-operational due to excessive sunlight etc. A typical driver can easily relate to this phenomenon and thereby would not lose confidence in the system. The use of a narrow notch filter, as well as polarizing filters, can significantly improve the separation of the artificially illuminated reflected light from the light reflected from the sun. Additionally, the camera shutter can by synchronized with the transmitted light.
Initially, one would assume that the only situation that the driver of a vehicle should be concerned with is if he or she decides to change lanes and after looking into the rear view mirror and not seeing an object in the blind spot, proceeds to change lanes. Unfortunately, the blind spot problem is significantly more complicated. The road may be curved and the lane changing maneuver might be quite easily accomplished. However, based on the geometry of the blind spot detecting system, using prior art systems, the driver is warned that he cannot execute such a lane change. This may be fallacious in that the vehicle that the system determines is in the blind spot may actually be in a different lane. Under the stress of congested driving conditions, the driver will not tolerate an erroneous message and thereby he might lose confidence in the system.
Identification of the object in the blind spot or other area of interest is important and a significant part of one or more of the present inventions disclosed below. Previous blind spot detectors have only indicated that there is a reflection from some object that is near the vehicle that may or may not interfere with the desired intentions of the vehicle operator to change lanes or execute some other maneuver. This is very disquieting to a vehicle operator who was told that something is there but not what that something is. For example, let us say that an operator of a vehicle wished to move that vehicle to the situation where he is partially on the shoulder in order to avoid a vehicle that is intruding onto his lane from the right. Most if not all current systems would tell the vehicle operator that he cannot do so. The system described in the present invention would say that there is a guard rail fifteen feet to your left, thereby allowing movement of 10 feet onto the shoulder and thereby avoid the vehicle intruding onto the lane from the right. This is a real world situation, yet all existing blind spot detection systems would give an erroneous answer or no answer at all to the vehicle operator.
Future automobile safety systems will likely be based on differential GPS and centimeter accurate maps of the roadway. The blind spot detector of this invention is an interim step to help eliminate some of the accidents now taking place. The particular geometry of the road is unknown to vehicles today, therefore, a blind spot detection system cannot use information that says, for example, that the road is about to take a sudden curve to the left, in its decision-making function. Nevertheless, this is a real situation and the system for detecting objects in the blind spot should not give erroneous information to be operator that he is about to have collision when the cause of this analysis is based on the assumption that the road will be straight when in fact a strong left turn is taking place. Note that prior to the presence of accurate maps even the inaccurate maps that now exist or of probe vehicle augmented maps can still be used to aid in this problem.
This problem cannot be solved absolutely at this time but if features such as angular position of the steering wheel of the host vehicle are data that can be entered into the system, then these types of situations can become less threatening. A preferred implementation of the present invention uses data from other vehicle sources in the decision making process including the steering wheel angle, vehicle speed etc. and map and location information if available.
In prior art blind spot detection systems, the inventor has generally realized that the operator of the vehicle cannot be continuously informed that there is an object in the blind spot. Every driver on the highway during rush hour would otherwise be subjected to a barrage of such warnings. Prior art systems have therefore generally provided an optical warning typically placed as an LED on the rear view mirror and an audible alert sounded when the driver activates the turn signal. Unfortunately, under normal driving conditions only about 70% of drivers use their turn signals as an indication of a lane change. Under stressful congested automobile driving situations, one can expect that the percentage would drop significantly. The driver must be warned when he is about to change lanes but the activation of a turn signal is not sufficient. Even crude maps that are available on route guidance systems today can add valuable information to the system by permitting the anticipation of a curve in the road, for example, especially if augmented with data from probe vehicles.
Various studies have shown that the intentions of a driver can sometimes be forecasted based on his activities during a several second period prior to execution of the maneuver. Such systems that monitor the driver and, using neural networks for example, try to forecast a driver's action can be expected to be somewhat successful. However, these computationally intensive systems are probably not practical at this time.
Another method is to provide a simulated audio rumble strip or vibrating actuation to the steering wheel at such time as the driver elects to redirect the motion of the vehicle based on an object in the blind spot. Whereas a rumble strip-type message can be sent to the driver, control of the vehicle cannot be assumed by the system since the road in fact may be executing a sharp curve and taking control of the vehicle might actually cause an accident.
The audio rumble strip method, or a tactile or other haptic messaging system, is the preferred approach to informing the driver of a potentially dangerous situation. Initially, a resistance would be applied to the steering wheel when the driver attempts to alter the course of the vehicle. Since the system will not know whether the driver is following a curve in the road or in fact changing lanes, the driver will be able to easily overcome this added resistance but nevertheless, it should indicate to the driver that there is a potential problem. If the driver persists, then a slight to moderate vibration would be applied to the steering wheel. Once again, this would be easily overcome by the driver but nevertheless should serve to positively warn the driver that he or she is about to execute a maneuver that might result in an accident based on the fact that there is an object in the blind spot.
The blind spot problem for trucks is particularly difficult. Trucks experience the same type of blind spot as do automobiles where the blind spot extends the length of the vehicle. However, the truck driver is also unable to see objects that are in another blind spot extending from forward of the front of the vehicle back typically 25 feet. This blind spot has been discussed in U.S. Pat. No. 5,463,384 and International Publication No. WO 90/13103. Trucks also have blind spots behind the trailer that are problematic during backup maneuvers. The invention disclosed herein is applicable to all three blind spot situations for trucks, automobiles or other vehicles.
It is noteworthy that some trucks have the capability of automatically rotating the side rear view mirrors based on the relative angle between the cab and the trailer. Such mirror systems are designed so that they maintain their orientation relative to the trailer rather than the cab. The blind spot monitoring system of this invention can make appropriate use of this technology to monitor the space along side of the trailer rather then cab.
Buses, trucks and various municipal people conveyors also have a blind spot directly in front vehicle and children have been run over by school buses when the driver was not aware that a child was crossing in front of the bus after embarking from the bus. The system of this invention is also applicable for monitoring this blind spot and warning a bus driver that a child is in this blind spot.
The images obtained from various locations outside of the vehicle can alternately be achieved by cameras or by fiber-optic systems. The inventions herein are not limited to the physical placement of cameras at particular locations when a fiber optic transmission system could be used as well.
1.6 Lane Departure Warning System
Various vehicle manufacturers are now offering early versions of a vision-based lane departure warning system on some vehicle models.
1.7 Night Vision
Various vehicle manufacturers are offering a night vision system on some vehicle models. Some of these systems are based on passive infrared radiation that is naturally emitted from warm bodies and is in the long wave or thermal region of the IR spectrum. Other manufacturers are offering active IR systems that operate in the near IR region of the spectrum that is just below the visual band in frequency. Despite claims to the contrary, it is the view of the inventor herein that such systems are of marginal value and may even contribute to degrading the safety of the vehicle since they can act as a distraction (as discussed below).
1.8 Headlight Control
Various systems have appeared from time to time to automatically dim the headlights of a vehicle when it senses the headlights of an oncoming vehicle. Such systems have now been removed from vehicle models since they had a large number of cases where the lights were dimmed when it was not necessary. Some examples were when the system sensed the reflection of the vehicle's own headlights from a sign, roadway furniture on a curve or even from the roadway when the road changes its angle at the start of a hill, for example. Such systems also did not dim the lights when the vehicle was following another vehicle.
A more sophisticated approach has been developed as described in U.S. Pat. No. 6,587,573. This patent makes use of pattern recognition techniques as disclosed in patents assigned to ATI to determine that light sources from vehicle's head or rail lights are present in images and are distinguishable from other sources of light. The solutions presented in this patent are, however, unnecessarily complicated and alternative approaches are disclosed herein.
2. Displays
Several systems have been proposed that display a view of the blind spot, using a video camera, onto a display either on the instrument panel or on the windshield as a “heads-up” display. Any system that displays a picture of the object on the screen that is inside the vehicle is also going to confuse the driver since he or she will not be able to relate that picture to an object such as another vehicle in the blind spot on the side of the host vehicle. Additionally, the state of the art of such displays does not provide equally observable displays at night and in bright sunlight. Thus, displays on a CRT or LCD are not natural and it is difficult for a driver to adjust to these views. The lighting of the views is too faint when sunlight is present and too bright when the vehicle is operating at night or the brightest object is not the object of interest and can be difficult to see in the presence of brighter objects. Therefore, none of the prior art television-like displays can replace the actual visual view of the occupant. In the discussion below, an icon display derivable from a pattern recognition system will be disclosed.
U.S. Pat. No. 6,429,789 discloses the use of icons in a display system for sensed exterior objects but does not explain how such objects are identified or how to effectively display the icons so as to not confuse the driver. To display an icon without knowing which icon to display or where to display it is of little value.
U.S. patent application Pub. No. 20040032321 describes use of a camera for viewing the space behind the vehicle for obstacles and a display showing the obstacles, but the display is a video screen and does not display icons. As a result, the display is difficult to see and interpret.
U.S. Pat. No. 5,949,331 describes a video display with the forecasted path of the vehicle overlaid. It does not identify other vehicles or objects nor represent them with icons.
An article on the EE Times website published Jan. 5, 2004, “Head-up displays get second glance”, describes a multicolor head-up display.
An Article in Nature, Issue 428, pages 911-918, Apr. 29, 2004 titled “The path to ubiquitous and low-cost organic electronic appliances on plastic” provides a good review of the state of the art for organic displays based on OLEDs which the inventor herein expects to be the future of automotive head-up displays.
European patent application EP1179958 describes use of an overhead view, as well as from any other direction, of a host vehicle along with a view of the objects that surround the vehicle. This differs from some of the ideas disclosed herein in that the invention actually compose a video image from up to eight camera images that are mounted on the vehicle exterior. No mention is made of the use of icons and this patent is a good example of the complexity of that approach and of the confusion that results especially when pixels which are not observable due to blockage are displayed as black. A preferred approach as disclosed herein is to identify objects that are in the vicinity of the host vehicle and then represent them as icons. This is a far simpler computational approach, results is a clearer image and allows for the full representation of the object including pixels that cannot be seen by the cameras. It also permits use of panoramic cameras thereby reducing the total number of cameras to four for imaging all areas surrounding the vehicle.
3. Identification
Neural networks and in particular combination neural networks are used in several of the implementations of this invention as the pattern recognition technology since it makes the monitoring system accurate, robust, reliable and practical. The resulting algorithm(s) created by a neural network program is usually only a few hundred lines of code written in the C or C++ computer language. The resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been expensive until recently, rendering their use for around vehicle area monitoring systems impractical. Similarly, the implementation of the techniques of the above referenced patents frequently requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities.
In using neural networks the data is usually preprocessed, as discussed below, using various feature extraction techniques. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. “Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets”, Neural Networks, Vol. 1. pp 75-89, 1988, and “Learned Classification of Sonar Targets Using a Massively Parallel Network”, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 and U.S. Pat. No. 5,517,667. Other examples include U.S. Pat. No. 5,235,339, U.S. Pat. No. 5,214,744, U.S. Pat. No. 5,181,254, and U.S. Pat. No. 4,881,270.
Although a trained neural network or combination neural network is contemplated for preferred embodiments of this invention, adaptive neural networks and other forms of artificial intelligent systems are also applicable especially where the host vehicle may be towing different loads that could confuse a static trained system. In this case, part of the system can be made adaptive to adjust to the particular load being pulled by the vehicle.
The field of neural networks is a developing field of research and the future can be expected to see vast improvements in artificial intelligence systems that go far beyond the concepts now in use of neural networks and associative memories. All applications of this work to solving automotive safety problems are contemplated herein and the term neural network will be used herein as representative of the class of such methods now in existence or to be developed. Of particular interest is a book by Jeff Hawkins, On Intelligence, 2004 Times Books, Henry Holt and Company, LLC, New York, N.Y. This book provides a basis for the development of future AI methods which will have applicability to automotive safety.
4. Anticipatory Sensors
The principles of the inventions disclosed herein can also be used for other purposes such as intelligent cruise control, speed over ground sensors, parking aids, height sensors for active suspensions, anticipatory crash sensors and obstacle detection systems. One particular application is for rear impact anticipatory sensors where both the distance and velocity (perhaps using Doppler principles where applicable) can be determined and used to deploy a movable headrest or equivalent.
The state of the art as of Apr. 23, 2001 of anticipatory sensors can be found summarized in a report by P. L. J. Morsink titled “Pre-crash sensing for increasing active and passive safety”, TON report) 1.OR.BV.013.0/PM which is available on the Internet at passivesafety.com. Most of the ideas on anticipatory sensing that appear in patents assigned to ATI and ITI covering anticipatory sensing that predate this report are not mentioned in the report or are handled superficially. Although mention is made of the need for object classification and other information, how to do it is generally missing from this report. The report lists the following information that is needed from sensors as:                time to impact        distance to the object        object classification (including size and shape)        trajectory of the obstacle        closing velocity        impact direction        acceleration of the incoming object        mass and stiffness of the incoming object        point of impact        
One point that is made with regard to anticipatory sensors is “The crash pulse is still necessary because, e.g., a radar sensor cannot distinguish an empty box or concrete pillar.” Although in principle if a concrete pillar looks like an empty box, this might be true, however the frequency of this happening is vanishingly small and in those and similar cases, the system would be biased not to set off the airbags for cases where an appropriate pattern recognition system is not sure of the identity of the object. In such cases, the default position would be to rely on the crash sensors. A properly trained pattern recognition system is not going to confuse a Mack truck traveling toward the vehicle at 60 mph with a cardboard box. Thus, the system discussed below will make the proper decision in 99+% of the cases and rely on the crash sensors whenever it is unable to make a decision. Due to this general misconception and teaching away from the inventions disclosed below, plus the 43,000+ roadway fatalities, there is clearly a long felt need for a proper and accurate anticipatory sensor as disclosed herein.
U.S. patent application Pub. No. 20050065688 also describes some of the ideas presented below and in ATI and ITI patents.
4.1 Positioning Airbags
Frontal impacts were the number one killer of vehicle occupants in automobile accidents with about 16,000 fatalities each year. Side impacts were the second cause of automobile related deaths with about 8,000 fatalities each year. The number of fatalities in frontal impacts as well as side impacts has been decreasing due to the introduction of airbags and mandatory seatbelt use laws.
Several automobile manufacturers are now using side impact airbags to attempt to reduce the number of people killed or injured in side impacts. The side impact problem is considerably more difficult to solve in this way than the frontal impact problem due to the lack of space between the occupant and the side door and to the significant intrusion of the side door into the passenger compartment which typically accompanies a side impact.
Some understanding of the severity of the side impact problem can be obtained by a comparison with frontal impacts. In the Federal Motor Vehicle Safety Standard (FMVSS) 208 49 kph crash test which applies to frontal impacts, the driver, if unrestrained, will impact the steering wheel at about 30 kph. With an airbag and a typical energy absorbing steering column, there is about 40 cm to about 50 cm of combined deflection of the airbag and steering column to absorb this 30 kph difference in relative velocity between the driver and vehicle interior. Also, there is usually little intrusion into the passenger compartment to reduce this available space.
In the FMVSS 214 standard crash for side impacts, the occupant, whether restrained or not, is impacted by the intruding vehicle door also at about 30 kph. In this case, there is only about 10 to 15 cm of space available for an airbag to absorb the relative velocity between the occupant and the vehicle interior. In addition, the human body is more vulnerable to side impacts than frontal impacts and there is usually significant intrusion into the passenger compartment. A more detailed discussion of side impacts can be found in a paper by Breed et al, “Sensing Side Impacts”, Society of Automotive Engineers Paper No. 940651, 1994.
Ideally, an airbag for side impact protection would displace the occupant away from the intruding vehicle door in an accident and create the required space for a sufficiently large airbag. Sensors used for side impact airbags, however, usually begin sensing the crash only at the beginning of the impact at which time there is insufficient time remaining to move the occupant before he is impacted by the intruding door. Even if the airbag were inflated instantaneously, it is still not possible to move the occupant to create the desired space without causing serious injury to the occupant. The problem is that the sensor that starts sensing the crash when the impact has begun is already too late, i.e., once the sensor detects the crash, it is usually too late to properly inflate the airbag.
There has been discussion over the years in the vehicular safety community about the use of anticipatory sensors so that the side impact accident could be sensed before it occurs. Prior to 1994, this was not practical due to the inability to predict the severity of the accident prior to the impact. A heavy truck, for example, or a tree is a much more severe accident at low velocity than a light vehicle or motorcycle at high velocity. Further, it was not possible to differentiate between these different accidents with a high degree of certainty.
Once a sufficiently large airbag is deployed in a side impact and the driver displaced away from the door and the steering wheel, he will no longer be able to control the vehicle that could in itself cause a serious accident. It is critically important, therefore, that such an airbag not be deployed unless there is great certainty that the driver would otherwise be seriously injured or killed by the side impact. Anticipatory sensors have previously not been used because of their inability to predict the severity of the accident. As discussed more fully below, the present invention solves this problem and therefore makes anticipatory sensing practical. This permits side impact airbag systems that can save a significant percentage of the people who would otherwise be killed as well as significantly reducing the number and severity of injuries. This is accomplished through the use of pattern recognition technologies such as neural networks such as discussed in U.S. Pat. No. 5,829,782.
Neural networks, and more recently modular neural networks, are capable of pattern recognition with a speed, accuracy and efficiency previously not possible. It is now possible, for example, to recognize that the front of a truck or another car is about to impact the side of a vehicle when it is one to three meters or more away even in fog, smoke, rain or snow and further away if range gating is used. This totally changes the side impact strategy since there is now time to inflate a large airbag and push the occupant out of the way of the soon to be intruding vehicle. Not all side impacts are of sufficient severity to warrant this action and therefore, there will usually be a dual inflation system as described below.
Although the main application for anticipatory sensors is in side impacts, frontal impact anticipatory sensors can also be used to identify the impacting object before the crash occurs. Prior to going to a full frontal impact anticipatory sensor system, neural networks can be used to detect many frontal impacts using data in addition to the output of the normal crash sensing accelerometer. Simple radar or acoustic imaging, for example, can be added to current accelerometer based systems to give substantially more information about the crash and the impacting object than possible from the acceleration signal alone.
The side impact anticipatory sensor of this invention can use any of a variety of technologies including optical, radar (including noise radar, Micropower impulse radar, and ultra wideband radar), acoustical, infrared or a combination of these. The sensor system typically contains a neural network processor to make the discrimination however a simulated neural network, a fuzzy logic or other algorithm operating on a microprocessor can also be used.
European Patent Publication No. EP0210079 (Davis) describes, inter alia, a radar system for use in connection with an airbag deployment apparatus to prevent injury to passengers when impact with an approaching object is imminent. Voltage level inputs representative of the distance between an object and the vehicle, the approach rate of the object with respect to the vehicle, the vehicle speed and driving monitor inputs, e.g., steering angles, turning rates and acceleration/deceleration, are all generated by appropriate detectors, weighted according to their importance to a normal vehicle operators' sensed safe or danger levels and then the weighted input voltages are summed to provide an “instantaneous voltage level”. This instantaneous voltage level is compared with a predetermined voltage level and if the instantaneous voltage level falls within a predetermined safe zone, output signals are not produced. On the other hand, if the instantaneous voltage level falls outside of the safe zone, i.e., within a danger zone, then the system can be designed to initiate deployment of the airbag on the additional condition that the vehicle speed is above a predetermined level. For example, the system can be programmed to deploy the airbag when the vehicle speed is between 35 and 204 miles per hour at a time of about 0.2 second prior to impact thereby enabling the airbag sufficient time to fully inflate.
Davis includes a radar system that includes an antenna assembly, a signal-processing unit and an output monitor. Davis relies on a radar signal generated by an antenna in the antenna assembly and which causes a return signal to be produced upon reflection of the radar signal against the approaching object. The return signal is received by a transceiver to be processed further in order to determine the distance between the object and the vehicle and the rate the object is approaching the vehicle. The return signal from the radar signal generated by the antenna is a single pulse, i.e., a single pixel. The elapsed time between the emission of the radar signal by the antenna and the receipt of the return signal by the transceiver determines the distance between the object and the vehicle and based on the elapsed time for a series of radar signals generated at set intervals, it is possible to determine the approach rate of the object relative to the vehicle.
In operation, the approach rate of the object relative to the vehicle, the distance between the object and the vehicle, the vehicle speed as well as other driving parameters are converted to voltage levels. Davis then uses an algorithm to weigh the voltage levels and compare the voltage levels to predetermined conditions for which airbag deployment is desired. If the conditions are satisfied by the results of the algorithm operating on the weighted voltage levels, then the airbag is deployed. In one embodiment, by appropriate manipulation of the voltage levels, false-triggering of the airbag can be prevented for impacts with objects smaller than a motorcycle, i.e., the voltage corresponding to a motorcycle at a certain distance from the vehicle is smaller than the voltage corresponding to a truck, for example at that same distance.
Davis does not attempt to recognize any pattern of reflected waves, i.e., a pattern formed from a plurality of waves received over a set period of time, from many pixels simultaneously (as with light and CCDs, for example) or of the time series of ultrasonic waves. A tree, for example can have a smaller radar reflection (lower voltage in Davis) than a motorcycle but would have a different reflected pattern of waves (as detected in the present invention). Thus, in contrast to the inventions described herein, Davis does not identify the object exterior of the vehicle based on a received pattern of waves unique to that object, i.e., each different object will provide a distinct pattern of reflected or generated waves. The radar system of Davis is incapable of processing a pattern of waves, i.e., a plurality of waves received over a period of time, and based on such pattern, identify the object exterior of the vehicle. Rather, Davis can only differentiate objects based on the intensity of the signal.
International Publication No. WO 86/05149 (Karr et al.) describes a device to protect passengers in case of a frontal or rear collision. The device includes a measurement device mounted in connection with the vehicle to measure the distance or speed of the vehicle in relation to an object moving into the range of the vehicle, e.g., another vehicle or an obstacle. In the event that prescribed values for the distance and/or relative speed are not met or exceeded, i.e., which is representative of a forthcoming crash, a control switch activates the protection and warning system in the vehicle so that by the time the crash occurs, the protection and warning system has developed its full protective effect. Karr et al. is limited to frontal crashes and rear crashes and does not appear to even remotely relate to side impacts. Thus, Karr et al. only shows the broad concept of anticipatory sensing in conjunction with frontal and rear crashes.
U.S. Pat. No. 4,966,388 relates to an inflatable system for side impact crash protection. The system includes a folded, inflatable airbag mounted within a door of the vehicle, an impact sensor also mounted within the door and an inflator coupled to the impact sensor and in flow communication with the airbag so that upon activation of the inflator by the impact sensor during a crash, the airbag is inflated.
U.S. Pat. No. 3,741,584 shows a pressurized air container and two air lines leading to a protective air bag. An air line passes through a first valve which is controlled by an anticipatory sensor and the other air line passes through a second valve controlled by an impact detector. The purpose of having two sensors associated with different valves is to ensure that the protective bag will inflate even if one of the crash sensors does not operate properly.
U.S. Pat. No. 3,861,710 shows an airbag inflation system with a single airbag which is partially inflated based on a signal from an obstacle detecting sensor and then fully inflated based on a signal from an impact detecting sensor. The obstacle detecting sensor controls release of gas from a first gas supply source into the gas bag whereas the impact detecting sensor controls release of gas from a second gas supply source into the gas bag. The first gas supply source includes a first gas container filled with a proper volume of gas for inflating the gas bag to a semi-expanded condition, a first valve mechanism, a pipe between the first gas container and the first valve mechanism and a pipe between the first valve mechanism and the gas bag. The second gas supply source includes a second gas container filled with gas in a volume supplementing the volume of gas in the first gas container so that the contents of both gas containers will fully inflate the gas bag, a second valve mechanism, a pipe between the second gas container and the second valve mechanism and a pipe between the first valve mechanism and the gas bag.
U.S. Pat. No. 3,874,695 shows an inflating arrangement including two inertia-responsive switches and coupled gas-generators. The gas-generators are triggered by the switches to inflate an airbag. The switches are both crash sensors and measured acceleration produced during the collision, and thus are not anticipatory sensors. The purpose of the two switches operative to trigger respective gas-generators is to enable the airbag to be inflated to different degrees. For example, if the crash involving the vehicle is a low speed crash, then only switch is actuated and gas-generated is triggered and the airbag will be inflated to part of its full capacity.
In U.S. Pat. No. 5,667,246, there are two accelerometers, each of which provides a signal when the value of the increase in deceleration exceeds a respective threshold value. The signal from the accelerometer is set to a first ignition stage and through a delay member to a second ignition stage. The second ignition stage also receives as input, a signal from the accelerometer and provides an inflation signal only when it receives a signal from both accelerometers. In operation, when the accelerometer sends a signal it serves to partially inflate the airbag while full inflation of the airbag is obtained only by input from both accelerometers.
Taniguchi (JP 4-293641) describes an apparatus for detecting a body moving around another body, such as to detect a car thief moving around a car. The apparatus includes a detection section supported on a support toll to the roof of the car. Taniguchi states that the detection section may be based on an infrared, microwave or ultrasonic sensor.
4.2 Exterior Airbags
Externally deployed airbags have been suggested by Carl Clark and William Young in SAE paper “941051 Airbag Bumpers”, 1994 Society of Automotive Engineers. Clark and Young demonstrated the concept of pre-inflating the airbag for frontal impacts but did not provide a method of determining when to inflate the airbag. The sensing problem was left to others to solve.
4.3 Pedestrian Protection
Although numerous patents have now appeared that discuss using an external airbag for the protection of pedestrians, little if any prior art exists on using anticipatory sensing for the detection of a pedestrian and for the pre-inflating of the pedestrian protection exterior airbag. Let us now consider some related art. None of the relevant ideas on the detection and classification of objects around the vehicle are believed to predate those disclosed in U.S. Pat. No. 6,343,810 and its linked predecessors.
Stereo cameras and neural networks are disclosed in a 1999 paper by L. Zhao and C. Thorpe at the itsc'99 Conference in Tokyo, Japan titled “Stereo- and Neural Network-Based Pedestrian Detection” and earlier in U.S. Pat. No. 6,370,475. The paper mentions that it can operate at video rate (30 frames per second) which is fast enough for automotive application; however, later they say that the stereo system can operate at 3-12 frames per second. Twelve frames would border on being too slow. The detection rate was stated at 85.2% with a false alarm rate of 3.1%. It is difficult to believe that such a system would be used for pedestrian detection for automobiles where hood raising or external airbags would be deployed with such a high false alarm rate. For such automotive applications, the false alarm rate would have to be effectively zero due to the large number of non-pedestrian scenes that a vehicle encounters on every trip. Other concepts discussed in the paper include:                Use of depth (from stereovision) segmentation to select candidate windows.        Normalization of candidate windows to 30×65 pixels.        Feed intensity gradient image directly into a feed-forward neural network.        Train using back-propagation and bootstrapping.        
Note that the performance is not high even with a small database and that depth segmentation will not work if pedestrians are not close to the camera.
T. Evgeniou and T. Poggio, in “Sparse Representation of Multiple Signals”, MIT AI Laboratory, September 1997, A.I. Memo No. 1619, attempt to fine sparse representations of a class of signals specifically those from images of pedestrians. In a sense, they use pattern recognition or it can be thought of as a deterministic determination. In any event, the recommend method is too complicated and too slow for practical use. Also, the accuracy of recognition of a pedestrian is not reported and it only considers forward facing pedestrians. This paper is related to pedestrian recognition only indirectly. It is a pattern-matching related approach, which has similar problems to those that have to be solved for Radial Basis Function (RBF) and Support Vector Machine (SVM) networks to reduce number of memorized support vectors.
This paper is very mathematical and is not closely related to pedestrian detection. The only application in the pedestrian detection problem is to help in the selection of a compact set of features from an over-complete transform such as wavelet decomposition.
C. Wohler, J. K. Anlauf, T. Portner and U. Franke, in “A time delay neural network algorithm for real-time pedestrian detection” 1998 IEEE International Conference on Intelligent Vehicles, pages 247-252, discuss a motion-based pedestrian detection system using stereo cameras which appears to also be too slow and too inaccurate. It uses neural networks in a time delay fashion which is similar to the combination neural networks developed by the current assignee. It is based on leg motion and therefore is not viable for pedestrians walking head-on into or away from an oncoming vehicle. It discusses using a fast stereo algorithm and motion tracking uses a Kalman filter but no data is provided as to how this is implemented.
C. Papageorgiou, T. Evgeniou and T. Poggio, in “A trainable pedestrian detection system”, Proceedings of Intelligent Vehicles, Stuttgart Germany October 1998, discuss a system that appears to not be based on a vehicle and even in its most advance form, the system requires 1.2 seconds to detect and classify a pedestrian and thus is too slow. Since the distance to a pedestrian is not recorded, it is difficult to make any judgment as to the accuracy of the system. This paper discusses the use of wavelets, edges and SVM, all of which have limitations as discussed herein.
The key point in this approach is feature extraction. Once the feature vectors are extracted, they can be processed using SVM, RBF, or simple feed-forward neural networks as has been done for years by the current assignee.
D. M, Gavrila, in “Pedestrian Detection from a moving vehicle”, Proceedings of the European Conference on Computer Vision, pp 37-49, Dublin, 2000, also reports on a pedestrian detection system. This system requires 1 second using a dual Pentium 450 megahertz processor. This is clearly too slow and requires too much processing power so as to render this system impractical. It makes use of template matching which is a slow computationally intensive pattern recognition method.
The technology in this article and in U.S. Pat. No. 6,556,692 by the same author focuses on finding the regions of interest or candidate windows in the images. The method is shape-matching (or correlation) against a preset template tree. The shape is found using edge detection, and fuzziness is added by distance transform or simply low pass filtering. In this paper, it is also proposed to use pattern matching approach based on texture features. FIG. 1(d) demonstrates picture after texture processing (DT). This is similar to some work done by the current assignee's employees. The inventor herein believes that pattern matching approaches should be avoided as they are very limited by artificially constructed pattern templates.
As discussed in U.S. Pat. No. 6,556,692 and in the current assignee's prior patents, when using neural networks approaches, “a priori knowledge” about the objects is not required. It is enough to have a properly marked training set with sufficient size. Modular neural networks are a better approach than the tree structure used in the '692 patent. Also, instead of softening with the distance transformation as discussed, a better approach is to use a “stop at an edge” technique, shifts of initial image and fixed set of networks in an ensemble, which can simulate “eye saccades”, and allows precisely defined and most probable place of an object in the image and increased the probability of a successful recognition.
U.S. patent application Pub. No. 20050084156 describes neural networks as something others might do. Since the effective use of neural networks, as taught in the current assignee's patents, is somewhat complicated. The fact that others might consider their use for pedestrian detection is not an enabling statement. The technique discussed uses a multi-path classification or detection:                Stereovision: obtain depth map and then run exhaustive search of the template database.        Image feature-based: use neural network to process features such as edge energy, intensity variation, symmetry, shadow, histogram, etc.        Include other data sources such as radar.The only detailed approach in this patent application is (histogram-based, no stereo, and only for separation between car and pedestrian):        1. Generate image histogram        2. Find a double threshold using contour score (with an improbable formula)        3. Generate binary image by applying double threshold        4. Generate row-sum and column-sum vectors        5. Calculate object score and apply threshold (using the same faulty formula)        
N. Checka, in “Fast pedestrian detection from a moving vehicle”, Proceedings of the 2004 Student Oxygen Workshop, MIT Computer Science and Artificial Intelligence Laboratory, uses cascade-based neural networks which is a subset of modular neural networks developed by the current assignee. No accuracy or speed data is provided in this paper. This paper is a very brief description. Nothing informative is presented except that the AdaBoost algorithm is used in a cascade architecture or multistage classification. There is no detailed explanation of the approach. In this paper some texture-like features appear to be used, actually wavelets, and some sort of modular neural networks. By comparison the assignee is using a wider range of features including edges and contours and has selected better texture features. There is nothing said about using sequences of images which can greatly improve the accuracy.
While reading Fang, Y., K. Yamada, Y. Ninomiya, B. K. P. Horn, & I. Masaki “A Shape-Independent Method for Pedestrian Detection with Far-Infrared Images,” IEEE Transactions On Vehicular Technology, Vol. 53, No. 5, September 2004, it becomes obvious that the authors of this paper, as well as all of the previous papers discussed above, do not really understand neural networks. Their application of neural networks is very primitive and doomed to failure. Also, this system is based on thermal infrared technology and it is very difficult to get the distance to an object based on thermal IR unless stereo cameras are used to which also is relatively inaccurate for longer separations. The accuracy of the system probably is poor since the distance to the pedestrian is not provided. At six frames per second, this system is also too slow.
The proposed method applies to night vision only. The detection consists of two stages: hypothesis and verification (a common approach to many detection/classification problems). The hypothesis (or segmentation) stage is done as follows:                1) Binarize far-infrared images        2) Count bright pixels at each column        3) Find peaks of pixel count along the horizontal direction        4) Within the regions of peaks, narrow down the vertical position based on brightness and intensity variation along the horizontal direction.The verification (or classification) stage is done by using (with or without a combination of) the following methods:        Similarity of the intensity histogram.        The fact that bright pixels usually locate near the center of the region of a pedestrian and often locate close to the boundary of the region of a non-pedestrian.        The fact that the pedestrian regions locate more closely in a data space defined by the vertical edge pixel counts in the region and its immediate upper/lower neighbors.        
A. Shashua, Y. Gdalyahu and G. Hayon. Pedestrian Detection for Driving Assistance Systems: Single-frame Classification and System Level Performance. Proc. of the IEEE Intelligent Vehicles Symposium (IV2004), June 2004, Parma, Italy, discloses a speed of ten hertz which is marginal. The accuracy is probably poor however since the accuracy is critically based on the distance from the vehicle and since this is not stated it is difficult on any of these papers to judge what the accuracy is when the pedestrian is no more than ten meters away. The method of dividing the pedestrian to separate segments is unnecessary if one understands the basis of neural networks and it is probably a poor choice in any event. Although it is difficult to ascertain, it looks like that for a distance of up to fifteen meters there is a ninety six percent accuracy with some false positives; however, it takes 4.6 frames to reach this accuracy which is approximately one half second which would be too slow. In the opinion of the inventor herein, the actual capability of generalization is not as good as it sounds in this paper, and the flat-road assumption can cause problems in the real world.
To summarize:
                The paper focuses on single-frame classification.        The single-frame classification algorithm has 2 stages:                    1. The region of interest is divided into sub regions, and orientation histogram (or edge energy) is extracted from each sub region. Then Ridge Regression is used to calculate the weights with which the best linear separation can be achieved within individual training clusters (i.e., subsets of training set).            2. Then, the inner-product of the weights and the feature values forms a new feature vector and AdaBoost is used to optimize against the entire training set.                        The paper lacks details on other modules of the entire system.        
The authors mention an “attention mechanism”, which generates about a fixed number of candidate windows (75 on average). The current assignee also employs an “attention mechanism” that generates arbitrary number of candidates depending on a particular image but usually the number is much smaller. The “attention mechanism” isn't described in the paper so it is difficult to evaluate it. The current assignee uses neural networks inside the “mechanism” to increase probability to find the Region of Interest (ROI) containing a pedestrian.
Similar to the described approach, the currently assignee also treats “Single Frame Classification Algorithm” (Part III) as one of most important parts of the system. Most important in the presented paper is splitting ROI to 9 sub-regions for independent analysis and manually preparing 9 separate training sets. These are strong limitations of the algorithm because it becomes suitable for pedestrians only. Also they use oriented gradients to construct a feature vector for recognition and (as it is described) some texture parameters to find a ROI. The presented approach is also limited. The assignee uses both types of features for recognition, as well as contours. The authors build their system on the 9 sub-regions instead of complex net for whole ROI. They mentioned that the results with a complex network, as taught by the current assignee, is better.
Several key features related to pedestrian recognition were first disclosed by ATI and ITI in the patents referenced herein. Among these ideas are the placement of the camera on the rear view mirror, the use of pattern recognition which includes template matching, segmentation, use of stereo or multiple cameras for monitoring the area of the space surrounding the vehicle, the general concept of detecting and classifying of pedestrians, vehicles, and other objects surrounding the vehicle, use of the visual, near infrared or far infrared portions of the electromagnetic spectrum and various advantages of using each of these particular portions of the spectrum.
In particular, the method of segmentation which is how to separate interesting objects in a scene so that they can be isolated and analyzed and identified separately using stereo vision, a scanning laser radar with distance measuring capabilities, and range gating were all believed to have been first disclosed by ATI and ITI in their patents. Several combinations of these techniques have been disclosed as examples of how one would combine various techniques. For example, a laser radar can be used in conjunction with a night vision far infrared system to measure the distance to the object emitting the radiation. In fact, using radar or any other method of detecting that there may be an object of interest including using the radiation emitted by that object which can be in the form of visual, thermal, or acoustic can be used to get the attention of system and subsequently cause a thin beam scanning laser radar to determine the location of emitting object. One technique which was not disclosed in detail but nevertheless obvious is to obtain a rough estimate of the distance of an object, and the object is classified for example as a pedestrian based on the size of the region of interest that has been isolated to contain the image of the pedestrian. Other obvious techniques are based on the vertical location of the object of interest in the camera field of view which may require the assumption that the road is relatively flat.
4.4 Rear Impact
The first disclosure of any rear impact sensor, either anticipatory or a contact sensor, is believed was made in U.S. Pat. No. 5,629,681 and U.S. Pat. No. 5,694,320. Such sensors can be used as disclosed in the current assignee's patents for deploying a variety of whiplash protection devices and the first disclosure of such a device deployed by a sensor was also made by the current assignee in the above patents. Although Japanese patent No. 2003-112545 is related, it is believed to follow the invention by the current assignee's inventors.
4.5 Positioning of Out-of-Position Occupants
U.S. patent application Ser. No. 11/423,596, now U.S. Pat. No. 7,338,069, describes positioning airbags and especially for such airbags that are triggered by an anticipatory sensor, and any prior art cited therein might be relevant to this aspect.
5. Agricultural Product Distribution Machines
A description of agricultural product distribution machines is set forth in the parent '598 application and in U.S. Pat. No. 6,285,938.
6. Distance Measurement
As discussed above, regardless of the distance measurement system used, a trained pattern recognition system, as defined below, can be used to identify and classify, and in some cases to locate, the illuminated object. Distance measurements by a variety of techniques can be used to determine the distance from the sensor to an object in the monitored area. They can also determine the distance to the ground for agricultural applications, for example, and provide correction information of the effective angle of transmission as will be discussed below.
Use of passive optical camera systems, such as the HDRC camera, has been discussed and the method of using either neural networks, optical correlation, or other pattern recognition systems has also been and will be discussed that illustrates how, in the present invention, the identity of the object occupying the area of interest will be determined. What follows now is a more detailed discussion of position determination.
For a preferred implementation of the system, the light from laser diodes will cross the field of view of the camera. If there is a heavy fog, for example, in the monitored area, then the reflection of the light off of the fog will create an elliptical image on the camera sensor array. This would also be true when heavy rain, smoke or heavy snowfall is present. This fact can be used to determine visibility. Observations of visibility conditions of objects in the area surrounding the vehicle even during severe weather conditions has led the inventor of this invention to the conclusion that when the visibility is so poor that the optical system using laser diodes described herein, for example, is not functioning with sufficient accuracy, that the operator of the vehicle should not be operating the vehicle on the roads and therefore the vehicle operator should be informed that safe travel is not possible. Thus, the use of radar or other technologies to view the blind spot, for example, which is actually quite close to the vehicle, is not necessary since vehicle operation should not be permitted when the visibility is so poor that the object cannot be seen in the blind spot, for example, by the systems of this invention. Nevertheless, the inventions herein can contribute to safe driving in these conditions, if such driving is attempted, since an indication will be obtained by the system based on the elliptical reflections from the laser diode indicating that the visibility is unacceptable. Note that when using a scanning IR laser radar system, the range of view of the system greatly exceeds that of the human operator especially when range gating is used to remove close-up reflections from the atmosphere (rain, snow, fog, smoke etc.)
For the embodiment of the invention using triangulation, it is desirable for the laser diodes, scanning laser diode or other light source to be displaced as far as reasonably possible from the camera in order to permit the maximum accuracy for the triangulation calculations. In an automobile, as much as six inches exists from one side of the exterior rear view mirror to the other side. This is marginal. For large trucks, the vertical distance separating the top and bottom of the rear housing can be as much as 24 inches. In both cases, the laser diode would be placed at one extreme and the camera at the other extreme of the mirror housing. An alternate approach is to place the camera on the mirror housing but to place the light source on the vehicle side. Alternately, both the camera and the light source can be placed at appropriate positions on the side of the vehicle. The key is that the direction of the light source should cross field of view of the camera at preferably a 10 degree angle or more.
Since the dots or a line created by a light source used to monitor the area of interest will likely be in the infrared spectrum and the majority of the light coming from objects in the monitored area will be in the visible spectrum, the possibility exists to separate them through the use of an infrared filter which will allow more accurately the determination of the location of the reflection from the laser diode onto the optical array. Such filters can be done either mathematically or through the imposition of a physical filter. However, this approach can require a mechanical mechanism to move the filter in and out of the camera field of view if visible light reception is also desired. Alternately, to eliminate the need to move the filter, a pin diode or equivalent dedicated receiver can be used to receive the reflected infrared light. Of course, multiple imagers can also be used, one for infrared and another for visible.
7. Scanners
A large number of patents and literature is available on rotating and vibrating mirror scanners and need not be listed here. These include rotating polygons such as used in surveying and office copiers and oscillating mirrors as in galvanometer type approaches. Scanners based on acousto-optical principles for use in automotive applications are new and are disclosed in the above-referenced patents and patent applications to ITI. A discussion on acousto-optic scanners can be found in U.S. Pat. No. 6,560,005.
8. Definitions
Preferred embodiments of the invention are described below and unless specifically noted, it is the applicant's intention that the words and phrases in the specification and claims be given the ordinary and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicant intends any other meaning, he will specifically state he is applying a special meaning to a word or phrase.
Likewise, applicant's use of the word “function” here is not intended to indicate that the applicant seeks to invoke the special provisions of 35 U.S.C. §112, sixth paragraph, to define his invention. To the contrary, if applicant wishes to invoke the provisions of 35 U.S.C. §112, sixth paragraph, to define their invention, he will specifically set forth in the claims the phrases “means for” or “step for” and a function, without also reciting in that phrase any structure, material or act in support of the function. Moreover, even if applicant invokes the provisions of 35 U.S.C. §112, sixth paragraph, to define his invention, it is the applicant's intention that his inventions not be limited to the specific structure, material or acts that are described in the preferred embodiments herein. Rather, if applicant claims his inventions by specifically invoking the provisions of 35 U.S.C. §112, sixth paragraph, it is nonetheless his intention to cover and include any and all structure, materials or acts that perform the claimed function, along with any and all known or later developed equivalent structures, materials or acts for performing the claimed function.
The definitions set forth in the parent '598 application, section 8 of the Background of the Invention section, are applicable herein.